Prajna: Cloud Service and Interactive Big Data Analytics

Apache Spark has attracted broad attention in both academia and industry. When people talk about Spark, the first thing that comes to mind is the Resilient Distributed Datasets (RDDs), which lets programmers perform in-memory computations on large clusters in a fault-tolerant manner. While RDD is certainly a great contribution, an overlooked aspect of Spark lies in its harness of functional programming concept (closure, lazy computation, and delegate) in distributed system building. We believe that this is what makes Spark flexible to support different data processing scenario under one common platform and to execute the distributed operation efficiently. We also believe that Spark has just scratched the surface, and the broader use of functional computing concept can fundamentally change how distributed system is built. In this paper, we describe Prajna, a distributed functional programming platform. Prajna is built on top of .Net and F#, and is open source released [31] (Apache license v2.0, note that all components that Prajna depends upon, including both .Net and F# have also been open sourced). Prajna not only supports (and extends) in-memory data analytics on large clusters like that of Spark, but also supports development and deployment of cloud services. Moreover, we show that Prajna can harmonize cloud service and data analytical service, and add rich data analytics on any existing cloud service/application. Prajna supports running of cloud service and interactive data analytics in both managed code and unmanaged code, and supports running of remote code with significant data components (e.g., a recognition model that is hundreds of megabytes in size). And with little programming effort (a day’s work), Prajna allows a programmer to add interactive data analytics to existing cloud services and applications with an analytical turnaround time under a second. The analyzed data (e.g., statuses of the cloud service, users’ inputs, etc.) is processed entirely in-memory during the analytical cycles and never stored to disk. Also, data is first locally accumulated and aggregated before further sending across the network for further cluster-wide aggregation. As such, through Prajna, cloud wide telemetry data becomes available right at users’ fingertip.